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IDENtlFIEHS • 
ABSTRACT 

The folloving lethods are suggested to Beasure the 
coaprehensiJjilitY of Children' is Television Workshop productions: (1) 
coiprehension tests iijiserted .into the program; (2) dudl audio- 
testing — a technique il»hich inserts comprehension questions into the 
empty spaces in the regular program audio track; and (3) program 
interruptions which pefimit sub jects to demonstrate comprehension in^ 
play or game situations\ In O'ther methods^ the subject indicates nben 
learning has occurred h% telling when he or she finds the apsver to a 
question asked before thfe program^ by narrating the program with only 
the audio or video portid^n as a recall aid^ by requesting repetition"^ 
of program segments^ or by audio responses when viewing. Another test 
measures the comprehensibility of stimulus material bj indexing the 
degree of distraction needi^d to make the^aterial incomprehensible. 
(EHH/PF) ' ^ 
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INTRODUCTION 

! ' - ' . ' ■ ' '/ ^ ' ^ . 

Given the assigned^ mission of formative research to seirve the 
needs of proojaction, approaches to comprehension measures will differ- 

! 

considerably from srtmmative research modelSj^ As presently und^stood, 
this means that moment-by-moment measures of comprehension are the 

■ ' . /r- " ^ ; - * . 

objective, so that elements of learning (beyond ^^^entioh) c^ be associated 
with the particular pr'ogramming techniques employed. This, in tufn, should 
.be helpfpl. to producers' in determining what programming concepts have or 
have net been educationally effective,/' 

Although analytically useful, it is recognizably distortion of the 
mu Iti-facGicd learning process to isolat^e comprehensibility for study* 
Palmer has listed four program attributes -that generate formative research 
at OrvU (1) ^appeal, (2) comj^rehensibility , (3) internal compatibility, and 
(4) activity eliciting potential. It is assumed that any one of these cate- 
gories can be affected by the status of the other three, and it Allows that 
formative research in areas other than comprehension can be related usefully 
t<^ comprehension research. In a separate ^document) a series* of proposals/ 
ideas will be presented for formative research in the area of program appeal. 

Palmer's definition df .comprehensibility will be employed: 



} - , . • ^ 

•Bie comprehensibility of a program or segment concerns 
the manner in which it is interpreted or construed by 
its viewers during the actual course of its presenta- 
tion — what they grasp of the intended instructional 
points, how they view the^, motives or. intentions of the 
characters.* 



The problems irivolved ^in devising moment-by-moment comprehension 
measures are formidable. They seem by their nature"^o require intrusion into 
the normal vi:n/ing process, as opposed tor the more unobtrusive techniques 
possible for measures of appeal/attention. The methodological objective, 
therefore, is tp pbtain a workable compa::omise. that permits sustained 
comprehension measures and sustained viewing without destroying^ either. 

Several ideas along these lifnes are described below. At this stage, our 

, " . , > " \ ■ 

major purpose is, to encc/urage discussion and constructive criticism of these 

suggestions, in the hope "bh^t some of them will warrant refinement and field 

x > ■ ■ , 

±est^.ng. . , 

^/ • . • 

I. SUBJECT IMDI'tgATES WHEN LEAF^ING HAS OCCURRED 

/ ■ > . . • 

0 ■ ■ 

t - ■ « 

In brief, the strategy employed here would be to ask a compre- 
hension or learning question before viewing, requesting the subject to 



^dicate when, during the program, he think^s he has the answer. 



Several conditions must be iaet: 



1. The question (or gcjestion^ must be\ answerable with a 

minimum of verbalization. 

\ • ' ; 

2. Tti6 subject must not forgetjwljile-'viev/ing that he\has a 

question to answer. Q^r thought here is that the "question 

\ ■ [ ' 
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could be presented (or re-presented) vi^u^ly,, perhaps 
by rear-sex eeh projection^ in a prominent manner tha,t 
would allow sustained awareness of the question • 

3, The subject should have a couple of s^plfe (and 
'reinforcing) practice trials with the procedure. 

4« Subjects should be pretested to ensure that they do not 
, kixow the answer before viewing. 

5. The self- indication of learning should be precisoly 
timed, so that these points |2cin later be associated 
precisely with program mciterial.. We suggest a. button 
that, when pushed, woultj stop the tape and automatically 
^ record the elaf)sed ^ime into the program. Subjects would* 
be "run" ona at a time*. 

Data presentation would/could bv as follows: 

Self-Reported Lea^'ning/Ccnprehonsibn Points 



in 



Number of 
Subjects 



n 



Minutes into 
Program 




Segment B, etc. 



It shoulh be realized .that this proceiiure would have highly - 
(Sensitized stjibjects who would be active and targeted information seekers. 
The natural viewing situation presxamably would elicit less learning or iess 
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rapid learning. • - 

To the extent that the instructional stimfllus. would require covert 
mental rehearsal before the learning "tbok," this procedure could chart this 

* 

delay tiiue between sti^nulus and response. Comparison of various delay ^ 
intervals could yield insights into the type of material best suited for 

follow-up to- "heavy" material. 

problems that should be anticipated and considered include howj^ ' 
tb handle "w.onJ" ^s^.ers and, "randdm" or "playful" button-pushing.- Suggestions 
on\ese and other 'problems would be most welcome. Also, suggestions- on how ; 
to co\ert this test to a "game" setting -would be useful.' 



i 



II. COMPREHENSION '"TESTS" INSE l^D INTO THE PROGPAM ^ •. 

In brief, this strategy would employ videotaped "tests," 

sn-^cific, prescloctcd learning objectives. th±t ^■rov.^d 
be inserted immediately after relevant program segir.ents. . 

^ AS presently envisioned, these tests would ask a question or pose 
a probl^ in a waf that could be "answered" with a simple behavior, such 
' as raiding a hand. This implies a version of the multiple' choice format 
for the test insertions. A preferable, alternative response procedure 
might involve a button-pushing and aromatic recording apparatus. This 
should reduce intersubject response influence in a group testing situation. 

m "natural" ppgrams or Specially composed pr^,grama where instruction 
redundancy is featured (exact or varied^ repetition) . repeated testing within 
the program would yield information on the optimum an^ount of redundancy. ^ 
An ideal condition might be to have the test inserts specially 

> _ 
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produced in an en1;ertainiog manner, so as to minimize the feeling of 
"interuptiorr" during the program, ^his is probably too expensive, however, 
and even "home-made" VTR test inserts should be less intrusive than 
repeatedly stopping the tape and having a 'researcher present the tests 
in persoj\. . . ' ^ 

This design 'is free from some of \the problems in Design I, e.g., 
• a series o^ questions cart: be asked (not just one or two) , and there should 
be no difficulty in having the subject temember what the question v/as.. This 
advantage comes at the cos^: pf not being able to associate learning with 

precise points within segments, bu.t only to learning displayed at the end 

t . ' . ■ < ' 

of segments. This m.ay or may not be sufficiently precise to be helpful to* 

pro<3ucefs. ! " . J , ' 

Data, presentation here would be^ very simple: 

* ' . ^ . V ~ • 

♦< 

/ - . o 

A. Segment A (description of objective ^and approach^ 

Test 'Insert A (question aslcedO* 

■ d 

% choosing or:tion A 



(N = ) B 



* correct option. 



B. Segment B (etc.) ^ 



III. DUAL AUDIO TESTING , ' , ■ ' dH. ' 

This suggestion is based on terry Boston's article oh dual aucjio 
television, his idea being, to su^lement ccsmnercial television programs with 
tin enrichment audio trkck, thus converting ahy program into a directed 
iearning experience. Here, in brief, the strategy^ would be to insert, as 
iiobtrusivQly as possible, f^irect questions into the regular apdib track 
without real time*^ alteration o£ the program. Again, as in Designs I and II, 



/ - w 

I 

a imiltiple choice fomat seems most feasible, as does pn automatic 'recording 
device that would allow gioup testing. 

There are several advantages to this design. , Production costs 
would be reasonable, and professional audib productioj. should he 'feasible 
(e.g., use of an interesting and distinctive voice which, through developed' 
convention, would comelto be Jfecognized as the •'^estion voico") . The normal 
flow of the program is \not interrupted, as it is in Design IT. ^In theory, 
the dual audio ques*Lon:; could be inserted at any point, withiti or following 
^program segments. i ^ ' > ' 

• ^ The^e are^l^o problems with, this design. A question requiring 

.visualization would obviously be impossible. The fast pacing of CTW*s 
productions may present a problem in finding enough regular pauses in ^ 
the action to insert a dual audio question ^nd get a response. If this 
proves to be the qase, the\ design would need alteration, probably going 
to a jfreoze frame on the video, and running the dual audio question for 
whatever -time- is needed. This v/ould Vield the additional possibility of 
referring questions directly to visual material on the screen. It is^ 
recognized that this mod?.fied yersion. functionally approximates the "stop- 
tape" formative research -done with "The Electric Goit^ny." Data presentation 
would be similar to that for Design II. ^ 

^ IV. PROGRAM INTERRUPTION FOR "PLAYFUL" TEST SITUATION 

This suggestion would be useful. only in those Situations where 
comprehension or learning could be displayed in a game or play situation. 
The idea is simply to -terminate the viewing after a critical segment has 
finished, and invite the subject (s) to play the "game" that will demonstrate 
^ whether the relevant learning took plorn d-.r-!-- -^^^ rcrm^nt. An . in all 



comprehension test suggestions so fat, subjec±S should be pretested to 

*~ ' • t> • ~ - . 

ensure that the learning v/as jibt acquired prior to yiev/ing. 

^ * - % • *' . " * 

V. SUBJECT NARRATES WJtH AIDED RECZALL ' ^ " • 

We note from Palmer^s •M^irch, 1972 paper that CTW is already employing 

this* idea to some extent. A subject is- shown, a program, then given only the 

audio or video portion of the program as recall aid while he relates the 

Btory (or answers specific questions). Some variations of this te-hnique 

*;...•'. « 

may be of intexest: ~ ^- . 

*. . • ' ^ . ■ 

: A. Key words or phrases in th6 audio could be deleted, with^the 

subject recjuest^d to vocalize th6 missing data. Special^ 
signalling devices; Cbells, U^hts,^ d'bq;^)^uld alert the 
eubject to an upcoming requeist for information. "Instant 
replays" could re-establish the cpn'text of the information 
as many times as would be required (this bear^ some, slight 
resemblance to Clo>^.e procecSure) . By its nature, such a 
procedure would allow precise location of learning within 
the program segment. , - \ 

B. '^a?here may be test situations where it would be desireabl-e 
to black out a portio n of the video, in addition to tlie audio. 
It is conceivable, for example, that "The Electric Company" 
video could be restricted primarily to printed words, with . 
all other "extraneous", material removed. This could help 
target the requested narration on specific learning objectives. 

C. A subject could be asked to "tell a friend" the "story" of 
a particular show, thus providing a non- testing rationale 
for th- narration, '-^-icli v cald then be examined for comprDhension. 



/ 



D. In -program segments having a plot progression or linear 
structure of cuiy typer the subject couJ.d be given a series 
of photographs taken off the scr6"en, asked to arifange them 
/ in "proper" order, and then to tell the* story around the 
photos. ' 

E* A small group of children could be asked to re-enact 

through role-playing a program segment. Observation and # 

I 

analysis of the interaction could index not only canprehension, 
bu-t also such factors as value^ perceptions/ As a variation, 
. ' a child's criticjue of a* specially taped children's re-enactanent 



could yiejra many of the same insights, 

r- 

VI, SUBJECT DETEI^IB^ S VVMOUNT OF EXPOSURE AFTER '^'EXAM SET" IS INDUCED 

The strategy here would be to indicate to the subject before 
exposure that several questions about the program will fee aske^ after the 
^'show, but that he can watch it as often as' he tikes before the quest ionning 
starts. Two playback units would bemused: one playing the regular program, 
the other always cued to the beginning of that segment. After each se*^ent, 
the siibject would be asked if he v.'anted 'to see that segment ' again or if he 
wanted to continue. As many repetitions as desired would be played. 

^^^i|ie skillful prpb?.ng at the end of segments/ as to why repetitions 
were or were not requested could give insight into the subject's reasoning, 
processes. . ' . ' . 

We have cpnsideirable intuitive doubts about how effective an "exam 
set" inducement might he for young children. Reactions from the research 
staff will be helpful on this point. . ' / 
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\ If some version of t^iis idea is Eventually feasible, it is -^antici- 
pated that the questions or. rationales -for requesting ' or not requesting tlie. 
, repetition would be more valuable " than the data on the distribution of. - V 
reque.sts themselves* ^ . 

vii^ ' audio monitoring of children's au dio behavior throughout the day 

! ■ 
The suggestion here S to equip a small sample of children in their 

' natyir^l home settings with wireless microphones that would transmit all their 

■ audio behavior to Voice- activated audio recorders for an extended per.iod of, 
1. . ■ I \ ■ - 

time, say, a, week. • 

There are -.two main advantages to this design: (A) it tests in the 
-actual setting, a condition rarely achieved; and (B) it provides data on 
•reactions during the program that should be related to many variables, ^ 
' including but not restricted to comprehension, and this program-reaction 
information can be interpreted in a "total environment" of behavior ior 
each child. Actual utilization^ of concepts learned frmCTW programs could 

be dejjionstrated. 

The analysis is essentially qualitative: it Would involve ^. 

- simple transcription of verbal data, and inferences would be made on the function 
and impact of programs from the child's point oE view. 

The "Big -.rother" overtones of suc;h a procedure call for extraordinary 
care and safeguards," as well' as responsibility in its i.i]^.lementatio^.. Ethical 
issues such as^ invasion, of privacy should be worked oUt befbre" even an • ^ 

- exploratory test ran is made. Hopefally, the richness a«d validity of 
• information pro^iided would make this preplanning worthwhile. 
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VIII, DEGP.ee of COHPREHENSIBILITY IM ED by THE ^MOUNT OF IE STRK.TION 
' WEEDED TO' ERODUCE iNCOMPr^gHENSIBILITY 

The strategy employed in this design is to test the degree of 

9^ ^ 11 

, . -9- " ' 
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comprehensibility of stimulus material by indexin^f the degree pf distraction 
("noise") needed to inake the material incomprehensible. The assumption is 
that material^that is most easily perceived is most easily comprehended and 
most easily learned! * The procedure thus measures a correlate of comprehension 
rather than direct measurement of comprehension, 

%■ ' • " , 

This design cam be employed to examyie.whol^^ bits or components 

within bits. ^ ^ ' . \ 

The fqHp|/ing procedures may ba utilized. ; , 

1. Ss view dm stimulus material in a .group viewing- situation. 

» AllvSs have earphones for audio channel of .the stimulus and \ 

"tape recorders <or an assortment of pictoral questionnaires) 

2. As Ss ,view pro<3rams, increasing amou1it$ of audio distraction 
(whirse, .buzz, etc.) or video distraction (snow^ ^distortion, \. , 
blackout, whiteout, etc.) are intrt?ducGd into the proper 
stimulus channel. ^ . 

3. Ss are requested to describe the occurrences on. the screen 

as they are happening (a running "narifative" or "shadowing" of 
the stimul.us material) . Three ways of sli^dow^ are here 
presented: * . . ,v 

.Ss tape record a "blow-*by-blow" description of this stimulus 
1 materials, Whetv s'l^^able to lontlnue intelligibly 

recdnstrujcting the entirel messag^, he is to begin describing 
anything whicH comes throvtoh the distortion. Whqn nothing 
* y comes through, he is to stQP all verbalization, 
b. Ss answer a series of simple, pictoral tests utilizing a 

multiple- choice format. ;. Questions asked would be: "Which 

■ *" . • * - ' ■ 

best dcscrilxis what is n . / - ^% Vou think . 

" ' > ** * ' « 

■ ' . • ' ■ , - •■ ■ . .• • 

iT 



you see nqw?^% etc. Tests woiild be conducted at frequent 



intervals cued by a ''bleep. During the test intervals^ 
distractions would/be stabilized, 

Ss are to "shado^?^ silently and push a button when compro- 
heiision af stimu-{us matelrialS diminishes vto an u^itelligibl< 
level* Then Ss/ are queried concerning the last scene 
' (elements) which they perceived and clearly ccanprehenijled. 
This ouestionning can' utilize either a verbal or visu^al ^ 
format. In either case^ a multiple-choici? approach appears 



. *o b6 ap^opriate. 

Much work will be needed to develop pictoral questionnaire^ 
approptlate.'to: tbe bit to be tested. Suggestiohs as tb the facilitation of 
this type of testing are needed. * ^ ^ 

As in most ot t^e designs suggested, stratification of the sazapie 
into age groups or Ndevelopraent-sta^fi groups; would yield more precise informa- 
tion. Either individual "oi gi®up testing is° feasible.' This design also_ 
created an approximation of the 'distracting: reception circumstances fox\much 
of the real world audience. 



